{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7.1.4 SVM for Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "简单的线性回归模型:\n",
    "$$\\frac{1}{2}\\sum_{n=1}^N{y_n-t_n}^2+\\frac{\\lambda}{2}\\lVert{w}\\lVert^2$$\n",
    "\n",
    "用不敏感误差函数替换平方损失函数:\n",
    "$$E_\\epsilon(y(x)-t)=\\begin{cases}0,&if|y(x)-t|<\\epsilon\\\\|y(x)-t|-\\epsilon&other\\end{cases}$$\\\n",
    "于是误差函数变为:\n",
    "$$C\\sum_{n=1}^NE_\\epsilon(y(x)-t)+\\frac{\\lambda}{2}\\lVert{w}\\lVert^2$$\n",
    "引入2个松弛变量:\n",
    "$$\\xi_n>0,if t_n>y(x_n)+\\epsilon$$\n",
    "$$\\hat\\xi_n>0,if t_n<y(x_n)+\\epsilon$$\n",
    "对应条件变为:\n",
    "$$t_n\\leq y(x_n)+\\epsilon+\\xi_n\\\\t_n\\ge y(x_n)-\\epsilon-\\hat\\xi_n$$\n",
    "支持向量的回归误差函数写成:\n",
    "$$C\\sum_{n=1}^N(\\xi_n+\\hat\\xi_n)+\\frac{\\lambda}{2}\\lVert{w}\\lVert^2$$\n",
    "Lagrange函数为:\n",
    "$$L(w,b,\\xi_n,\\hat\\xi_n)=C\\sum_{n=1}^N(\\xi_n\\hat\\xi_n)+\\frac{\\lambda}{2}\\lVert{w}\\lVert^2-\\sum_{n=1}^N(\\mu_n\\xi_n+\\hat\\mu_n\\hat\\xi_n\\\\-\\sum_{n=1}^Na_n(\\epsilon+\\xi_n+y_n-t_n)-\\sum_{n=1}^N\\hat a_n(\\epsilon+\\hat\\xi_n-y_n+t_n)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\frac{\\partial L}{\\partial w}=0\\Rightarrow w=\\sum_{n=1}^N(a_n-\\hat{a}_n)\\phi(x_n)\\\\\n",
    "\\frac{\\partial L}{\\partial b}=0\\Rightarrow \\sum_{n=1}^n(a_n-\\hat a_n)=0\\\\\n",
    "\\frac{\\partial L}{\\partial \\xi_n}=0\\Rightarrow a_n+\\mu_n=C\\\\\n",
    "\\frac{\\partial L}{\\partial \\hat\\xi_n}=0\\Rightarrow \\hat a_n+\\hat\\mu_n=C$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对偶表示为:\n",
    "$$\\tilde L(a,\\hat a)=-\\frac{1}{2}\\sum_{n=1}^N\\sum_{m=1}^N(a_n-\\hat a_n)(a_m-\\hat a_m)k(x_n,x_m)-\\epsilon\\sum_{n=1}^N(a_n+\\hat a_n)+\\sum_{n=1}^N(a_n-\\hat a_n)t_n$$\n",
    "\n",
    "限制条件为$$0\\leq a_n\\leq C\\\\0\\leq \\hat a_n\\leq C$$\n",
    "$$y(x)=\\sum_{n=1}^N(a_n=\\hat a_n)k(x,x_n)+b$$\n",
    "KKT条件为:\n",
    "$$a_n(\\epsilon+\\xi_n+y_n-t_n)=0\\\\\\hat a_n(\\epsilon+\\hat\\xi_n+y_n-t_n)=0\\\\\n",
    "(C-a_n)\\xi_n=0\\\\(C-\\hat a_n)\\hat\\xi_n=0$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ⾸先，我们注意到如果$\\epsilon + \\xi_n + y_n − t_n = 0$那么系数an只能⾮零，这表明数据点要么位于ϵ-管道的上边界上($\\xi_n=0$)，要么位于上边界的上⽅($\\xi_n>0$)\n",
    "* $\\hat\\xi_n$的⾮零值表示$\\epsilon + 、\\hat\\xi_n + y_n − t_n = 0$，这些点必须位于ϵ-管道的下边界上或者下边界的下⽅。\n",
    "* $\\epsilon$-管道内部的点$a_n=\\hat a_n=0$\n",
    "\n",
    "$$b=t_n-\\epsilon-w^T\\phi(x_n)=t_n-\\epsilon-\\sum_{n=1}^n(a_m-\\hat a_m)k(x_n,x_m)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
